Variational Method
This page contains resources about Variational Methods and Variational Bayesian Inference. Subfields and Concepts * Variational Calculus / Calculus of Variations * Variational Analysis‎ * Variational free energy * Free energy principle * Reducing / Controlling the variance ** Control variates ** Rao–Blackwellization ** By linear regression * Evidence Lower Bound (ELBO) * Structured Variational Inference * Kullback–Leibler (KL) Divergence * Variational Bayes * Variational Bayesian EM (VBEM) * Stochastic Variational Inference * Stochastic Gradient-based Variational Inference * Stochastic Gradient Variational Bayes (SGVB) Estimator * Variational Autoencoder * Expectation Propagation ** Loopy Belief Propagation / Sum-Product Message Passing * Kullback-Leibler Variational Inference / Mean field Variational Bayes ** Structured Mean field / Structured Variational Approximation ** Weighted Mean Field * Tree-based reparameterizations * Tree-reweighted belief propagation * Bethe and Kikuchi free energy * Online Bayesian Variational (OBV) Inference Algorithms * Non-conjugate Variational Inference * Automated Variation Inference ** Black-Box Variational Inference (BBVI) ** Automatic Differentiation Variational Inference (ADVI) ** SGVB with local expectation gradients (LeGrad) ** SGVB with reparametrization-based gradient (ReGrad) ** SGVB with the log derivative trick (LdGrad) * Stochastic Optimization * Energy-Based Model (EBM) ** Free energy (i.e. the contrastive term) ** Loss functionals or Loss functions *** Energy Loss *** Generalized Perceptron Loss *** Generalized Margin Losses *** Negative Log-Likelihood Loss Online Courses Video Lectures *Graphical Models and Variational Methods by Christopher Bishop - VideoLectures.NET *Approximate Inference by Tom Minka - VideoLectures.NET *Machine Learning: Variational Inference by Jordan Boyd-Graber *Variational Inference by Chieh Wu *Autoencoding Variational Bayes by Durk Kinga - ICLR 2014 *Variational Autoencoders by Karol Gregor Lecture Notes *COS597C: Advanced Methods in Probabilistic Modeling BY David M. Blei *Lecture: Variational Inference by Russ Salakhutdinov Books and Book Chapters * Bengio, Y., Goodfellow, I. J., & Courville, A. (2016). "Chapter 19: Approximate Inference". Deep Learning. MIT Press. * Theodoridis, S. (2015). "Chapter 13: Bayesian Learning: Approximate Inference and Nonparametric Models". Machine Learning: A Bayesian and Optimization Perspective. Academic Press. * Murphy, K. P. (2012). "Chapter 21: Variational inference". Machine Learning: A Probabilistic Perspective. MIT Press. * Barber, D. (2012). "Section 7.7: Variational Inference and Planning". Bayesian Reasoning and Machine Learning. Cambridge University Press. * Barber, D. (2012). "Chapter 11: Learning with Hidden Variables". Bayesian Reasoning and Machine Learning. Cambridge University Press. * Barber, D. (2012). "Chapter 28: Deterministic Approximate Inference". Bayesian Reasoning and Machine Learning. Cambridge University Press. * Koller, D., & Friedman, N. (2009). "Chapter 11: Inference as Optimization". Probabilistic Graphical Models. MIT Press. * Bishop, C. M. (2006). "Chapter 10: Approximate Inference". Pattern Recognition and Machine Learning. Springer. * MacKay, D. J. (2003). "Chapter 33: Variational Methods" Information Theory, Inference and Learning Algorithms. Cambridge University Press. Scholarly Articles * Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2016). Variational inference: A review for statisticians. arXiv preprint arXiv:1601.00670. * Mandt, S., Hoffman, M. D., & Blei, D. M. (2016). A Variational Analysis of Stochastic Gradient Algorithms. arXiv preprint arXiv:1602.02666. * Titsias, M., & Lázaro-Gredilla, M. (2015). Local expectation gradients for black box variational inference. In Advances in Neural Information Processing Systems (pp. 2638-2646). * Archer, E., Park, I. M., Buesing, L., Cunningham, J., & Paninski, L. (2015). Black box variational inference for state space models. arXiv preprint arXiv:1511.07367. * Hoffman, M. D., & Blei, D. M. (2015). Structured stochastic variational inference. In Artificial Intelligence and Statistics. * Salimans, T., & Knowles, D. A. (2014). On using control variates with stochastic approximation for variational Bayes and its connection to stochastic linear regression. arXiv preprint arXiv:1401.1022. * Ranganath, R., Gerrish, S., & Blei, D. M. (2014). Black Box Variational Inference. In AISTATS (pp. 814-822). * Lázaro-Gredilla, M. (2014). Doubly stochastic variational Bayes for non-conjugate inference. In International Conference on Machine Learning. * Salimans, T., & Knowles, D. A. (2013). Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis, 8''(4), 837-882. * Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013). Stochastic variational inference. ''Journal of Machine Learning Research, 14(1), 1303-1347. * Wingate, D., & Weber, T. (2013). Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299. * Fox, C. W., & Roberts, S. J. (2012). A tutorial on variational Bayesian inference. Artificial intelligence review, 38(2), 85-95. * Paisley, J., Blei, D., & Jordan, M. (2012). Variational Bayesian inference with stochastic search. arXiv preprint arXiv:1206.6430. * Wainwright, M. J., & Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning, 1''(1-2), 1-305. * Tzikas, D. G., Likas, A. C., & Galatsanos, N. P. (2008). The variational approximation for Bayesian inference. ''IEEE Signal Processing Magazine,25(6), 131-146. * Wainwright, M., & Jordan, M. (2005). A variational principle for graphical models. New Directions in Statistical Signal Processing, 155. * Beal, M. J. (2003). Variational algorithms for approximate Bayesian inference. Ph.D. Dissertation, University College London. * Xing, E. P., Jordan, M. I., & Russell, S. (2003). A generalized mean field algorithm for variational inference in exponential families. In Proceedings of the Nineteenth conference on Uncertainty in Artificial Intelligence (pp. 583-591). Morgan Kaufmann Publishers Inc. * Wainwright, M. J., & Jordan, M. I. (2003). Variational inference in graphical models: The view from the marginal polytope. In Proceeding of Annual Allerton Conference of Communication Control and Computing (Vol. 41, No. 2, pp. 961-971). * Lawrence, N. D. (2001). Variational inference in probabilistic models. ''Ph.D. Dissertation, University of Cambridge. * Minka, T. P. (2001). ''A family of algorithms for approximate Bayesian inference. Ph.D. Dissertation'', Massachusetts Institute of Technology. * Ghahramani, Z., & Beal, M. J. (2001). Propagation algorithms for variational Bayesian learning. ''Advances in neural information processing systems, 507-513. * Attias, H. (2000). A variational Bayesian framework for graphical models.Advances in neural information processing systems, 12(1-2), 209-215. * Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999). An introduction to variational methods for graphical models. Machine learning,37(2), 183-233. Tutorials *Challenges in Variational Inference: Optimization, Automation, and Accuracy by Rajesh Ranganath - NIPS 2015 *Variational Auto-Encoders and Extensions by Durk Kingma - NIPS 2015 *Stochastic Backpropagation, Variational Inference, and Semi-Supervised Learning by Durk Kingma - NIPS 2014 *Auto-Encoding Variational Bayes by Durk Kingma - 2014 *Auto-Encoding Variational Bayes by Durk Kingma (Video) - ICLR 2014 *Stochastic Gradient VB. Intractable posterior distributions? Gradients to the rescue! by Durk Kingma - 2014 *Speeding up Gradient-Based Inference and Learning in deep/recurrent Bayes Nets with Continuous Latent Variables by Durk Kingma - 2014 *Variational Bayesian inference by Kay H. Brodersen - 2013 *High-Level Explanation of Variational Inference by Jason Eisner - 2011 *Graphical models and variational methods by Martin Wainwright - ICML 2008 *Variational Methods by Zubin Ghahramani - 2003 *Variational Mean Field for Graphical Models by Baback Moghaddam Software * Vilds - Black box variational inference for state space models in Python * Edward: A library for probabilistic modeling, inference, and criticism - Python with TensorFlow * VIBES * VBA toolbox - MATLAB See also *Monte Carlo Methods, an other set of methods for Approximate Inference in Graphical Models *Ensemble Learning *Estimation Theory *Information Theory *Control Theory Other Resources * Variational-Bayes - A repository of research papers, software, and links related to the use of variational methods for approximate Bayesian learning up to 2003 * The lure of free energy - Blog post Category:Probabilistic Graphical Models